Add Linux ARM64 (aarch64) wheel support #415

aoirint · 2023-10-23T12:22:17Z

This pull request is taking over #382. Thanks @imWildCat ( #382 (comment) )!

Add Linux ARM64 (aarch64) wheel to build_wheels.py and fix some platform dependent code.

Here is the main points of fixes.

Python wheel platform tag need to be aarch64 (PEP 600)（the 3rd return value of bdist_wheel.get_tag(), setup.py#L82）
Keep the .so filename in bastibe/libsndfile-binaries as arm64 (PYSOUNDFILE_ARCHITECTURE, setup.py#L24).
The possible arm64 compatible architecture list ["aarch64", "aarch64_be", "armv8b", "armv8l"] is referred to this StackOverflow answer (platform.machine() returns uname -m value on Linux).

With only this pull request, pytest is not passed due to an issue of libsndfile_arm64.so distributed in bastibe/libsndfile-binaries. Please see the GitHub Issue below.

Linux ARM64 binary libsndfile_arm64.so is not compiled with libogg and libvorbis libsndfile-binaries#29

Currently, this pull request does not update the submodule _soundfile_data to keep the reference for bastibe's repository.
Updating submodule after bastibe/libsndfile-binaries#29 fixed is required for working properly either before or after merging this pull request.

Here is an experimental ARM64 (aarch64) binary wheel build using my own libsndfile_arm64.so binary ( bastibe/libsndfile-binaries#29 fixed one). This binary successfully passes pytest.

https://github.com/aoirint/python-soundfile/releases/tag/0.12.2.dev2%2Baoirint.addlinuxarm64
- Download soundfile-0.12.2.dev2+aoirint.addlinuxarm64-py2.py3-none-manylinux_2_17_aarch64.whl

Full log of pytest using the experimental wheel (successfully passed)

$ python -m pytest
================================================= test session starts ==================================================
platform linux -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0
rootdir: /code
collected 324 items

tests/test_argspec.py ....                                                                                       [  1%]
tests/test_soundfile.py ........................................................................................ [ 28%]
................................................................................................................ [ 62%]
................................................................................................................ [ 97%]
........                                                                                                         [100%]

=================================================== warnings summary ===================================================
tests/test_soundfile.py::test_file_truthiness[obj]
  /home/user/.local/lib/python3.11/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning: Exception ignored from cffi callback <function SoundFile._init_virtual_io.<locals>.vio_write at 0x5518f63ec0>: None

  Traceback (most recent call last):
    File "/home/user/.local/lib/python3.11/site-packages/_pytest/fixtures.py", line 596, in _get_active_fixturedef
      return self._fixture_defs[argname]
             ~~~~~~~~~~~~~~~~~~^^^^^^^^^
  KeyError: 'request'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/user/.local/lib/python3.11/site-packages/_pytest/fixtures.py", line 599, in _get_active_fixturedef
      fixturedef = self._getnextfixturedef(argname)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/user/.local/lib/python3.11/site-packages/_pytest/fixtures.py", line 473, in _getnextfixturedef
      raise FixtureLookupError(argname, self)
  _pytest.fixtures.FixtureLookupError: ('request', <SubRequest 'file_w' for <Function test_file_truthiness[obj]>>)

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/code/soundfile.py", line 1261, in vio_write
      written = file.write(data)
                ^^^^^^^^^^^^^^^^
  ValueError: I/O operation on closed file

    warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================ 324 passed, 1 warning in 3.49s ============================================

If you are using Docker, you may test my ARM64 (aarch64) wheel build/fix on x86_64 machine with QEMU CPU emulation.

Commands

Docker Hub: arm64v8/python

# On host machine
git clone https://github.com/aoirint/python-soundfile.git
cd python-soundfile
git checkout 5f2f5b37cd210005f0510f3d11cd9e4e0efa06e4
git submodule update --init

sudo docker run --rm -it --volume ".:/code" --platform "linux/arm64" arm64v8/python:3.11 bash

# As root user in Docker container
useradd --uid 1000 --create-home user
su -l user -s /bin/bash

# As general user in Docker container
cd /code

# If you want to build your own wheel
python build_wheels.py

# Please replace the wheel file name for your own wheel
pip install ./dist/soundfile-0.12.2.dev2+aoirint.addlinuxarm64-py2.py3-none-manylinux_2_17_aarch64.whl

pip install numpy pytest
python -m pytest

…ux-aarch64

bastibe · 2023-10-25T15:08:00Z

Awesome to see progress on this! The changes so far look entirely reasonable.

Are you working on the aarch64 build scripts over in libsndfile-binaries as well?

aoirint · 2023-10-25T17:12:24Z

Are you working on the aarch64 build scripts over in libsndfile-binaries as well?

We don't need any new build script for libsndfile-binaries. It has already been merged in the pull request below.

Adds CI job for building linux arm64 library (using qemu) libsndfile-binaries#24

It seems you just need to run the GitHub Actions workflow manually, then upload/push the Linux ARM64 build artifact libsndfile.so as libsndfile_arm64.so into bastibe/libsndfile-binaries (overwrite existing libsndfile_arm64.so).

bastibe · 2023-10-27T16:57:54Z

Thank you for the heads-up! I probably won't have time to look into this in the next few days, but next weekend or so looks promising.

bastibe · 2023-12-10T11:41:53Z

I have just updated libsndfile-binaries to the newest version of libsndfile, which should include support for Vorbis in ARM64 in #423.

However, in order to test these things, we'd need a test suite that runs uses the precompiled binaries, in addition to the platform-supplied libsndfile. Would you, by any chance, like to contribute such a test suite? I'm afraid I don't have enough time to do that in my spare time at the moment.

aoirint · 2023-12-10T16:41:45Z

Thanks for creating the updated shared libraries.

Would you, by any chance, like to contribute such a test suite?

OK. I will try work on it.

Add test for Linux environment using libsndfile from OS package
- Remove _soundfile_data before installing soundfile package

Also, python-soundfile is not currently tested for the ARM64 platform. We should test this if possible.

Add test for Linux ARM64 environment emulated on Linux x64

However, it is complicated to test ARM64 on GitHub Actions because there are only the x64-native CI runners.
We might need to add a separated CI job instead of adding arm64 to strategy.matrix.architecture.

We have some options. If you don't mind, I prefer to use Docker because it is easier to implement for me.

Docker multi-platform image
Use pguyot/arm-runner-action@v2 like the build job of libsndfile_arm64.so
- https://github.com/bastibe/libsndfile-binaries/blob/27341b2ec76a7ccccb2826e76dc938159bfb1370/.github/workflows/build-libs.yml#L106-L127

Example implementation using Docker:

docker run --name "emulator" --volume ".:/code" --platform "linux/arm64" --detach arm64v8/python:3.11 tail -f
docker exec "emulator" useradd --create-home user
docker exec "emulator" apt-get update
docker exec "emulator" apt-get install -y libsndfile1
docker exec -u user "emulator" pip install numpy pytest cffi>=1.0
docker exec -u user "emulator" python soundfile_build.py
docker exec -u user -w /code "emulator" pip install --editable . --verbose
docker exec -u user -w /code "emulator" python -m pytest .

bastibe · 2023-12-12T06:56:32Z

OK. I will try work on it.

Thank you very much! This is highly appreciated!

Also, python-soundfile is not currently tested for the ARM64 platform. We should test this if possible.

Indeed, that is a problem. The use of docker for this is absolutely fine. Thank you for bringing this up, I wasn't aware of that option. Do you want to address this as part of this PR, or in a new PR?

Out of curiosity, does it run faster than the pguyot/arm-runner-action@v2-action? IIRC, that build step was very slow compared to the others in libsndfile-binaries.

imWildCat and others added 4 commits April 7, 2023 19:08

Add linux arm64 to build_wheels.py

02172bd

Merge remote-tracking branch 'imWildCat/add-linux-arm64' into add-lin…

4b82661

…ux-aarch64

linux arm64: use pep600 architecture name for platform tag

c8912c5

arm64: use packaged libname "arm64" on any arm64-compatible machine

5f2f5b3

aoirint mentioned this pull request Oct 23, 2023

Add linux arm64 to build_wheels.py #382

Closed

aoirint changed the title ~~Add Linux ARM64 (aarch64) wheel~~ Add Linux ARM64 (aarch64) wheel support Oct 23, 2023

Merge remote-tracking branch 'origin/master' into add-linux-arm64

8c3f52b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Linux ARM64 (aarch64) wheel support #415

Add Linux ARM64 (aarch64) wheel support #415

aoirint commented Oct 23, 2023

bastibe commented Oct 25, 2023

aoirint commented Oct 25, 2023 •

edited

Loading

bastibe commented Oct 27, 2023

bastibe commented Dec 10, 2023

aoirint commented Dec 10, 2023 •

edited

Loading

bastibe commented Dec 12, 2023

Add Linux ARM64 (aarch64) wheel support #415

Are you sure you want to change the base?

Add Linux ARM64 (aarch64) wheel support #415

Conversation

aoirint commented Oct 23, 2023

bastibe commented Oct 25, 2023

aoirint commented Oct 25, 2023 • edited Loading

bastibe commented Oct 27, 2023

bastibe commented Dec 10, 2023

aoirint commented Dec 10, 2023 • edited Loading

bastibe commented Dec 12, 2023

aoirint commented Oct 25, 2023 •

edited

Loading

aoirint commented Dec 10, 2023 •

edited

Loading